Description:

In conversation_parts, we attempt to only yield parts that have been updated since the previous sweep. Because of this, we can only checkpoint after checking a full date window. If a date window contains a large number of updated conversations, it can take conversation_parts more than one day to check all those conversations for updated parts. This has been observed for at least one task; about 4,000 pages of conversations were updated within 1 day (the smallest configurable date window).

This commit adds some dynamic reduction logic to size of the date window conversation_parts checks in a single fetch_conversation_parts invocation. If there are more than 25 pages of conversations, then the date window is split in half until there are <= 25 pages or the date window is the smallest possible size (1 second).

An alternative approach would be to yield all parts of an updated conversation, then conversation_parts can checkpoint after each page of conversations. I chose to do dynamic window reduction since that's keeps the connector's current behavior of only yielding updated parts, but this alternative approach could be worth while later if the date window approach is too slow & we aren't extremely concerned about yielding duplicate, non-updated parts.

Workflow steps:

(How does one use this feature, and how has it changed)

Documentation links affected:

(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)

Notes for reviewers:

Tested on a local stack. Confirmed the date window for a single invocation of fetch_conversation_parts is reduced until there are <= 25 pages of conversations or the date window is the smallest possible size of 1 second.

This change is